Well-typed generic smart-fuzzing for APIs

Despite recent advances in program certification, testing remains a widely-used component of the software development cycle. Various flavours of testing exist: popular ones include unit testing, which consists in manually crafting test cases for specific parts of the code base, as well as QuickCheck-style testing, where instances of a type are automatically generated to serve as test inputs. These methods of testing can be thought of as internal testing: the test routines need to access the internal representation of the datastructures that are used by the functions under test. They can also be thought of as per-function testing: a test suite is built (by hand, or automatically) for each function that must be tested. We propose a new method of external testing that applies at the level of the module interface. The core of our work is a small embedded domain specific language to describe APIs, i.e., functions and data-types. Then, these API descriptions are used to drive the generation of test-cases. We have successfully used this method in two different contexts: Test case generation. First, we implemented a library dubbed ArtiCheck that combines the functions exported by a given module interface to build elements of the various data-types exported by the module, and then checks that all the elements of these data-types meet user-defined invariants. Smart fuzzing. Second, the first author reimplemented this methodology while working at Cryptosense to automate the analysis of (security) APIs. More precisely, Cryptosense’s Testing library uses an API description to automatically exercise vendors’ implementations of the said API.